Procedure and device for detecting fades in a sequence of images

ABSTRACT

The invention relates to a method for detecting a fade in a sequence of images which comprises the following steps:
         calculating, for each of the images of a window N consecutive images in length, termed the detection window, and for the image preceding the detection window, a luminance level on the basis of the luminance values associated with each of the pixels of each of the images;   calculating, for each of the N images of the detection window, a difference of luminance level between the luminance level of the current image and the luminance level of the image preceding the current image; and   deducing therefrom the presence of a fade for the detection window if, for at least k images of the detection window, said difference of luminance level is greater that a predefined threshold, with 2≦k≦N.

1. FIELD OF THE INVENTION

The invention relates to a device and a procedure for detecting fades ina sequence of images.

2. STATE OF THE ART

In the field of image processing (e.g. compression, contentenhancement), the methods applied to image sequences, for example toestimate a motion between two images or else to temporally filter theimages of the sequence, often require temporal tracking of theinformation. But a break in temporal continuity in a sequence of imagesdisrupts the implementation of these methods. This more or less sharpbreak corresponds to a more or less significant modification (e.g.transition) of the content between two successive images. Itsidentification is necessary so as to take account of this discontinuityduring the application of these methods and thus improve theirrobustness.

More particularly, a shot or sequence shot is an uninterrupted sequenceof images filmed by a camera or a camcorder in a single picture capture.This shot is the basic unit used to analyse or construct a videocontent. A video sequence generally comprises a succession of shotsinterconnected with the aid of video editing procedures. Theseprocedures in particular make it possible to create an abrupt transitionbetween two shots or else a progressive transition according to which animage is progressively changed into another image by mixing of the twoimages (e.g. a fade). The detection of these transitions makes itpossible in the case of an image sequence coding application to improvethe coding quality by taking account of these transitions in the choices(for example structure of a group of images or “GOP”) made by the coder.This detection also makes it possible to retrieve the shot boundariesand narrative units so as in particular to allow non-linear navigationaround the content.

3. SUMMARY OF THE INVENTION

The invention is aimed at detecting progressive transitions, moreparticularly fades, in a sequence of images.

The invention relates to a method for detecting a fade in a sequence ofimages comprising pixels or image points with each of which isassociated at least one luminance value. It comprises the followingsteps:

-   -   calculating (10), for each of the images of a window N        consecutive images in length, termed the detection window, and        for the image preceding the detection window, a luminance level        on the basis of the luminance values associated with each of the        pixels of each of the images;    -   detecting (11) a fade, for each of the N images of the detection        window, based on the difference between the luminance level        associated with the current image and the luminance level        associated with the image preceding the current image; and    -   deducing therefrom (12) the presence of a fade for the detection        window if, for at least k images of the detection window, a fade        has been detected, with 2≦k≦N.

Preferably, the luminance level associated with an image is equal to thesum of the luminance values associated with each of the pixels of theimage.

Advantageously, the method furthermore comprises an additional step forvalidating the presence of a fade in the detection window if a level ofmotion associated with the detection window is less than a predeterminedthreshold.

Preferably, the level of motion associated with the current image isequal to the ratio between the temporal activity calculated between thecurrent image and the image preceding the current image, termed thepreceding image, and the spatial activity calculated for the currentimage. Moreover, the level of motion associated with the decision windowis equal to the level of motion associated with the last image of thedecision window.

Preferably, the temporal activity between the image and the precedingimage is equal to the sum of the absolute values of the pixel-wisedifferences between the luminance values associated with each pixel ofthe image and the luminance values associated with each pixel of thepreceding image. Furthermore, the spatial activity of the image is equalto the sum of the absolute values of the luminance values associatedwith each pixel of the image.

The invention also relates to a device for detecting fades in a sequenceof images comprising pixels or image points with each of which isassociated at least one luminance value. It comprises:

-   -   means (32, 33, 34) of calculation for calculating, for each of        the images of a window N consecutive images in length, termed        the detection window, and for the image preceding the detection        window, a luminance level on the basis of the luminance values        associated with each of the pixels of each of the images;    -   means (32, 33, 34) of detection for detecting a fade, for each        of the N images of the detection window, based on the difference        between the luminance level associated with the current image        and the luminance level associated with the image preceding the        current image; and    -   means for deducing (32, 33, 34) the presence of a fade for the        detection window if for at least k images of the detection        window a fade has been detected by the detection means, with        2≦k≦N.

The invention relates furthermore to a video coding device whichcomprises coding means and a device for detecting fades according to theinvention.

The invention relates moreover to a device for structuring a sequence ofimages characterized in that it comprises indexation means and a devicefor detecting fades according to the invention.

4. LISTS OF FIGURES

The invention will be better understood and illustrated by means ofwholly non-limiting advantageous exemplary embodiments and modes ofimplementation with reference to the appended figures in which:

FIG. 1 illustrates a method for detecting fades according to theinvention;

FIG. 2 illustrates a position of a decision window;

FIG. 3 illustrates a displacement of a decision window;

FIG. 4 illustrates another displacement of a decision window;

FIG. 5 illustrates a device according to the invention;

FIG. 6 illustrates an image sequence coding device according to theinvention; and

FIG. 7 illustrates an image sequence structuring device according to theinvention.

5. DETAILED DESCRIPTION OF THE INVENTION

The invention is aimed at detecting fades in a sequence of images, thesequence being progressive or interlaced, each image comprising pixelswith each of which is associated in particular a luminance value. In thecase of interlaced sequences, each image comprises an even field and anodd field. Each field itself comprises pixels or image points with eachof which is associated in particular a luminance value. Considering asequence of images and a sequence of images f (for example a sequence ofblack or nearly uniform images), a fade from f to s is called a fade-inand vice versa a fade from s to f is called a fade-out. The sequence ofimages comprising the fade from s to f or from f to s is defined asfollows:

e(x,y,n)=(1−x(n))·s(x,y,n)+x(n)·f(x,y,n)

where:

-   -   x(n) is the fade factor,    -   n is the index of the image in the sequence, and    -   (x,y) are the coordinates of the pixels in the image.        The factor x(n) is homogeneous inside an image or frame (i.e.        independent of x and y) and lies between 0 and 1. We define the        luminance level L_(e)(n) for a given image or frame n in the        following manner:

${L_{e}(n)} = {{\sum\limits_{x = 0}^{X}{\sum\limits_{y = 0}^{Y}{e\left( {x,y,n} \right)}}} = {\sum\limits_{x = 0}^{X}{\sum\limits_{y = 0}^{Y}\left\lbrack {{\left( {1 - {x(n)}} \right)*{s\left( {x,y,n} \right)}} + {{x(n)}{f\left( {x,y,n} \right)}}} \right\rbrack}}}$

It follows that

${L_{e}(n)} = {{\left( {1 - {x(n)}} \right){\sum\limits_{x = 0}^{X}{\sum\limits_{y = 0}^{Y}{s\left( {x,y,n} \right)}}}} + {{x(n)}{\sum\limits_{x = 0}^{X}{\sum\limits_{y = 0}^{Y}{f\left( {x,y,n} \right)}}}}}$

and therefore that L_(e)(n)=(1−x(n))*L_(s)(n)+x(n)L_(f)(n). By assumingthat the image f(x,y,n) is homogeneous, has a luminance level equal toL_(f) and that L_(f)(n)=L_(f)(n+1), then the difference between theluminance level associated with image or frame n and the luminance levelassociated with image or frame n+1 is equal to:

L _(e)(n+1)−L _(e)(n)=(1−x(n+1))*L _(s)(n+1)−(1−x(n))*L_(s)(n)+(x(n+1)−x(n))L _(f)

Considering that: L_(s)(n+1)−L_(s)(n)=ΔL_(s)(n+1), the luminancedifference is then equal to:

L _(e)(n+1)−L _(e)(n)=ΔL _(S)(n+1)*(1−x(n))−(x(n+1)−x(n))*(L _(s)(n+1)−L_(f))

If the fade factor x(n) is linear for a fade which lasts N images orframes from an image (or frame) of index 0 to an image (or frame) ofindex N then x(n) is split up in the following manner:

${x(n)} = {k_{1} + {k_{2}\frac{n}{N}}}$

k₁ and k₂ are two coefficients making it possible to express the linearnature of the fading factor. In the case of a fade-in: x(0)=1 andx(N)=0. In the case of a fade-out: x(0)=0 and x(N)=1. It follows thatthe variation in the luminance level between images n and n+1 is equalto:

${{L_{e}\left( {n + 1} \right)} - {L_{e}(n)}} = {{\Delta \; {L_{s}\left( {n + 1} \right)}*\left( {1 - k_{1} - {k_{2}\frac{n}{N}}} \right)} - {\frac{k_{2}}{N}*\left( {{L_{s}\left( {n + 1} \right)} - L_{f}} \right)}}$

The variation in the luminance level between two successive images (orframes) of the sequence s is generally negligible with respect to theluminance variation between the sequence s and the sequence f, therebymaking it possible to simplify the equation above as follows:

${{L_{e}\left( {n + 1} \right)} - {{Le}(n)}} \approx {{- \frac{k_{2}}{N}}*\left( {{L_{s}\left( {n + 1} \right)} - L_{f}} \right)}$

By making the assumption of the stationnarity of the luminance levelalong the sequence s, the equation simplifies in the following manner:

${{L_{e}\left( {n + 1} \right)} - {{Le}(n)}} \approx {{- \frac{k_{2}}{N}}*\left( {L_{s} - L_{f}} \right)}$

The latter assumption is valid most of the time insofar as most fadesoccur on the basis of a sequence of weak motion. Thus during a fade, thevariation in the luminance level between two images (or frames) isdependent on the variation in the luminance level between the start andthe end of the fade (L_(s)-L_(f)), and is inversely proportional to theduration (N images or frames) of the fade. The method according to theinvention is therefore based on the detection of a variation in theluminance level between two successive images. This variation isthereafter compared with an experimentally defined threshold.

According to a preferred embodiment, illustrated by FIG. 1, the methodis split up into 4 steps referenced 10 to 13. In this figure, themodules represented are functional units, which may or may notcorrespond to physically distinguishable units. For example, thesemodules or some of them may be grouped together in a single component,or constitute functionalities of one and the same software. A contrario,certain modules may possibly be composed of separate physical entities.Hereinafter in the document, to simplify the description the term imageis employed to designate either an image composed of two frames or oneframe alone.

Step 10 consists in calculating, over a window N images in length,termed the detection window, a luminance level for each image of thedetection window and for the image of the sequence which precedes thiswindow. As defined previously, the luminance level of an image is thesum over the whole of the image of the luminance values associated witheach of the pixels of this image. A window N images in length is aportion of the sequence comprising N successive images.

Step 11 consists in detecting a fade, for each image of the detectionwindow, by comparing the variation in the luminance level between thisimage, termed the current image, and the image which precedes it with athreshold. This threshold is a multiple of the size of the image as anumber of pixels. For example, it is equal to 1.75 multiplied by thewidth and by the height of the image as a number of pixels. If thevariation in the luminance level is greater than this threshold then afade is detected for the current image. Depending on whether one isseeking to detect fade-in or fade-out, the variation in the luminancelevel is calculated differently. Thus, in the case of a fade-in, thevariation in the luminance level is equal to the difference between theluminance level of the image preceding the current image and theluminance level of the current image. In the case of a fade-out, thevariation in the luminance level is equal to the difference between theluminance level of the current image and the luminance level of theimage preceding it. This step makes it possible to detect, for eachimage of the detection window, fade-ins and fade-outs separately.Advantageously, different thresholds can be defined so as to detect thefade-ins and the fade-outs.

Step 12 consists in deducing the presence of a fade for the detectionwindow on the basis of the detections performed in step 11. Moreprecisely, this step consists in calculating the number of images of thedetection window for which a fade has been detected. During this step,the results arising from the previous step for fade-in and fade-out arecombined thereby making it possible to detect a fade whether it be afade-in or fade-out. For this purpose, if a fade-in or fade-out has beendetected in step 11 for at least k (2≦k≦N) images of the detectionwindow, we deduce therefrom the presence of a fade for the detectionwindow. The length of the detection window is preferably equal to 8 andk is equal to 6. This step in particular makes it possible to avoiddetermining false fades due to noise, or else to an instability of thebrightness in the course of the sequence.

Advantageously, during an optional step 13, the result of the previousstep 12 is combined with a local evaluation of the level of motion ofthe decision window. This makes it possible to avoid overdetections onportions of sequence of images with strong motion that might possibly bewrongly regarded as fade. This level of motion can be provided forexample by an image level decision module (for example decision of theframe/image mode of coding, decision of the type of the image I, P, or B. . . ) of a coding device. It can be calculated for an image bycomputing the ratio between the temporal activity calculated betweenthis image and the image preceding it (for example the sum of theabsolute values of the differences, pixel-wise, between the luminancevalues associated with each pixel of this image and the luminance valuesassociated with each pixel of the image preceding it) and the spatialactivity calculated for this image (for example the sum over the imageof the absolute values of the luminance values associated with eachpixel of this image). More precisely, the level of motion calculated forthe last image of the decision window is the level of motion associatedwith the decision window. If the presence of a fade has been deduced forthe detection window at the previous step 12 and if the level of motionfor this window is less than a certain threshold, then the presence of afade for the detection window is validated.

In the particular case of an interlaced sequence, it is advantageous todefine a sliding window, termed the decision window. This window can forexample have a size of three frames. This window shown in FIG. 2 can inparticular be used, in a coding device, by an image level decisionmodule. In FIG. 2, the window is positioned so as to take an image leveldecision for the image referenced P_(k) corresponding to the framesreferenced F_(i−1) and F_(i−2). More precisely, the image P_(k) iseither an image consisting of the union of the two frames F_(i−1) andF_(i−2) (“frame picture”), or an image consisting of two independentframes F_(i−1) and F_(i−2) (“field picture”). In the case of the MPEG-2standard defined in the document ISO/IEC 13818-2 (entitled “Informationtechnology—Generic coding of moving pictures and associated audioinformation: Video”), the image level decisions (for example decision ofthe “frame picture”/“field picture” mode of coding, decision of the typeof the image I, P, or B . . . ) apply at least over two successiveframes. In the particular case of a sequence having undergone atransformation of 24-image adjustment type (“detelecine” or “3:2pulldown”), a frame is repeated. The size of the decision window istherefore three frames. In order to tie the fade detection to the imagelevel decisions, the variation in the level of luminance calculated instep 11, is calculated between the frame positioned in F_(i−1) in thesliding window and the frame at the same position in the sliding windowsuch as positioned so as to take an image level decision for thepreceding image referenced P_(k−1). FIGS. 3 and 4 represent the positionof the decision window for the following image P_(k+1). Specifically, ifP_(k) is composed of the union of F_(i−1) and F_(i−2) and if F_(i) isnot a repeated frame or if P_(k) is composed of two independent framesF_(i−1) and F_(i−2), then the following position of the decision windowfor the image P_(k+1) is illustrated by FIG. 3. On the other hand, ifP_(k) is composed of the union of F_(i−1) and F_(i−2) and if F_(i) is arepeated frame (“detelecine” case), then the following position of thedecision window for the image P_(k+1) is illustrated by FIG. 4.

The present invention also relates to a device for detecting fades,referenced 30 in FIG. 5, implementing the method described previously.Only the essential elements of the device are represented in FIG. 5. Thedevice 30 comprises: a random access memory 32 (RAM or similarcomponent), a read only memory 33 (hard disk or similar component), aprocessing unit 34 such as a microprocessor or a similar component andan input/output interface 35. These elements are linked together by anaddress and data bus 31. The read only memory 33 contains the algorithmsimplementing steps 10 to 12 and optionally step 13 of the methodaccording to the invention. On power-up, the processing unit 34 loadsand executes the instructions of these algorithms. The random accessmemory 32 in particular comprises the programs for operating theprocessing unit 34 loaded on power-up of the appliance, as well as theimages to be processed. The input/output interface 35 has the functionof receiving the input signal (i.e. the source image sequence) andoutputs the result of the fade detection according to steps 10 to 12 (or13) of the method of the invention.

The present invention applies to the field of image sequencecompression. Specifically, the device for detecting fades 30 can be usedin a coding device referenced 40 in FIG. 6. This device comprises inparticular coding means 41 for coding the source image sequence 400. Thecoding means comprise at least decision means making it possible toselect the type of image (e.g. intra (I image), predicted (P images),bidirectional (B images)). This module can for example implement theMPEG-2 coding standard defined in particular in the document ISO/IEC13818-2 (entitled “Information technology—Generic coding of movingpictures and associated audio information: Video”). More generally, itcan implement any standard for coding image sequences. In particular ituses the information provided by the device 30 to dynamically adapt thetype of image. This can make it possible to improve the cost ofcompression and the quality of the decoded images.

The device for detecting fades 30 can also be integrated with a devicefor structuring image sequences, referenced 50 in FIG. 5. This device 50comprises in particular indexation means 51 making it possible to createa description 501 of the source image sequence 400 (e.g. temporalmarker, otherwise known as a “time code”, of start and end of fades).This device 50 in particular makes it possible to retrieve the shotboundaries and narrative units so as to make it possible to navigatenon-linearly through the sequence or to generate digests. Moreprecisely, it makes it possible to retrieve the appropriate shotboundaries when a fade has been introduced during editing by virtue ofthe device for detecting fades 30 and therefore provides better basesfor structuring into narrative units. Moreover the knowledge of thefades gives high-level information on the structuring of the video. Thedevice 50 therefore makes it possible to structure the sequence forexample into chapters and sub-chapters in the case of digital sequencesof images as on a DVD. Specifically, in a film, a fade to black orfade-out followed by a fade from black or fade-in very often indicates achange of “chapter”. Moreover, in the sports retransmissions, slowmotions—which make it possible to detect the interesting phases of agame—are very often delimited by fades.

Of course, the invention is not limited to the exemplary embodimentsmentioned above. In particular, the person skilled in the art canintroduce any variant to the embodiments set forth and combine them tobenefit from their various advantages.

1. A method for detecting a fade in a sequence of images, said imagescomprising pixels or image points with each of which is associated atleast one luminance value, said method comprising the following steps:calculating, for each of the images of a window N consecutive images inlength, termed detection window, and for the image preceding saiddetection window, a luminance level on the basis of the luminance valuesassociated with each of said pixels of each of said images; calculating,for each of said N images of said detection window, a difference ofluminance level between the luminance level of the current image and theluminance level of the image preceding said current image; and deducingtherefrom the presence of a fade for said detection window if, for atleast k images of said detection window, said difference of luminancelevel is greater than a predefined threshold, with 2≦k≦N.
 2. A methodaccording to claim 1, wherein the luminance level of an image is equalto the sum of the luminance values of each of said pixels of said image.3. A method according to claim 1, that it comprising an additional stepfor validating the presence of a fade in the detection window if a levelof motion associated with said detection window is less than apredetermined threshold.
 4. A method according to claim 3, wherein thelevel of motion associated with an image is equal to the ratio betweenthe temporal activity calculated between said image and the imagepreceding said image, termed the preceding image, and the spatialactivity calculated for said image and in that the level of motionassociated with said decision window is equal to the level of motionassociated with the last image of said decision window.
 5. A methodaccording to claim 4, wherein said temporal activity between said imageand said preceding image is equal to the sum of the absolute values ofthe pixel-wise differences between the luminance values associated witheach pixel of said image and the luminance values associated with eachpixel of said preceding image and in that the spatial activity of saidimage is equal to the sum of the absolute values of the luminance valuesassociated with each pixel of said image.
 6. A device for detectingfades in a sequence of images comprising pixels or image points witheach of which is associated at least one luminance value, wherein itcomprises: means of calculation for calculating, for each of the imagesof a window N consecutive images in length, termed the detection window,and for the image preceding said detection window, a luminance level onthe basis of the luminance values associated with each of said pixels ofeach of said images; means of calculation for calculating for each ofsaid N images of said detection window, a difference of luminance levelbetween the luminance level of the current image and the luminance levelof the image preceding said current image; and means for deducing thepresence of a fade for said detection window if for at least k images ofsaid detection window said difference of luminance level is greater thana predefined threshold, with 2≦k≦N.
 7. A video coding device, comprisingcoding means and a device for detecting fades according to claim
 6. 8. Adevice for structuring a sequence of images, comprising indexation meansand a device for detecting fades according to claim 6.